This invention relates to the generation of an image of a three-dimensional object by application of texture to a three-dimensional (3D) model of the object.
Of particular interest to the applicant is the virtual representation of a human head and, throughout this description, reference will be made to a 3-D model of a human head. However it will be appreciated that the technique can be applied to a 3-D model of any object.
The requirement for a possible virtual representation of a human head was identified over ten years ago, and development work resulted in a head being represented in software by a model in conjunction with texture mapping of facial features onto the head. An example of such a technique is described in Welsh W J, Simons A D, Hutchinson R A and Searby S, xe2x80x9cSynthetic face generation for enhancing a user interfacexe2x80x9d, Proceedings of Image Com 90, Bordeaux, France pp 177-182 November 1990. Together with techniques to compress video data, the head was shown to require sufficiently low bandwidth to be able to be demonstrated effectively over Ethernet-type LANs, and also over the Internet. A user can rotate the head in all three dimensions to provide an all-round viewing capability.
To a layman user of the head, the technique appears to be elegantly simple. A user has simply to type in the words he wishes to be spoken as plain text and, after a few seconds delay, the head speaks the words, reproducing the precise lip movements as if a human speaker had spoken the text. To add to the realism, the head blinks at the user as it talks, and its teeth and tongue can also be seen to be moving. Overall the impression is one of a computer generated head speaking as if it were an everyday occurrence, and it can be very convincing. To give an indication of the actual complexity involved, however, one can look at the hardware requirements to run the head effectively. Currently, for example, the head requires a dedicated 3-D graphics multi-processor machine (the Silicon Graphics Onyx) the size of two large filing cabinets to run effectively.
The anticipated uses for a virtual 3-D animated head are many, and the examples given below are certainly not exhaustive. The primary business use for the head is in the field of virtual-conferencing. The bandwidth requirements of such a system are much lower than that required for conventional video-conferencing, even with the use of sophisticated video data compression techniques. By having the software that providies the basic models already installed at both ends of a link, then upon initialisation of a virtual video-conference the only information that will need to be sent is that required to actually conform the head to look like a specific individual. Once this conformation process has been performed, the only data that need be sent over the network is the control information to produce the speech and the animation. Since the control information for the speech generation is plain text, it can be seen that the bandwidth requirements are drastically reduced. The talking head is likely to find application in multi-party virtual-conferences.
The head may also be used as a training tool for people who are hard of hearing. As the lip region of the head may be modelled very accurately, it is possible for a skilled person to lip read the head if necessary. Also, by observing the lip shapes that the head produces for words that the person has typed in, and by then attempting to copy the shapes in front of a mirror, it is proposed that the head may be of value in speech training.
A third use for the head is as an agent interface for on-line services. Science fiction books and films have for decades presented us with the vision of a computer controlled human representation selling us products and services from the comfort of our own homes. Such an agent interface may encompass a broad spectrum of on-line services such as home shopping, booking holidays, negotiating video-on-demand, acting as on-line help for web-based applications etc. The uses of such a virtual human representation are endless. By providing an instantly recognisable image, the head will help to humanise the computer interface, and so make interacting with a system more pleasant.
In the case of a model representing a head, it is necessary for a generic 3-D model to be conformed to the shape and size of the individual to be modelled. This may be a two stage process involving firstly changing the basic shape of the head to match the basic shape of the individual subject concerned and secondly texturing the now conformed head model with appropriate photographs of the individual in order to provide the appropriate detail required for recognition of the textured head image as a specific person. Note that the word xe2x80x9ctexturexe2x80x9d is used here in the specialised sense used in the image-processing art, namely the two-dimensional detail of a portion of a surface element (e.g. the luminance and colour of picture elements), not in the sense of implying a fine three-dimensional structure. The applying of texture (that is, creating an association between a surface element of a 3-D model and a corresponding portion of a 2-D image such that the latter is employed in producing a visual display of the object represented by the model) is referred to as xe2x80x9ctexturingxe2x80x9d. The shape conformation can be performed manually or by an automatic process. Such shape conformation is necessary to allow recognition of the modelled head of an individual as the individual it is meant to be. If the head shape is not conformed and a texture map containing the photographs of the individual is simply stretched to fit over the generic model, then odd effects occur and recognition may be lost. The shape must be conformed to represent the same ratios of, for example, head width to height, distance between the eyes to head width etc. for appropriate recognition of the head model as the individual subject.
Following the shape conformation process, the head is textured with appropriate photographs of the subject. This involves taking several photographs of the subject from different aspects i.e. front, side, under chin, back etc. and combining these into a single combined texture map.
As mentioned, known systems allow a user to manipulate the texture map by hand onto the model. When dealing with a 3-D model as complex as a head, however, the time required to texture the model by hand has hitherto been excessive, being heavily labour intensive, and taking anything from two days to more than a week, depending upon the quality of the photographs that were being used as the texture map.
According to one aspect of the present invention there is provided a method of generating an image representing a three-dimensional object, the three-dimensional object being modelled as a stored set of parameters representing a model of a three-dimensional object and at least two two-dimensional images of the object, each image representing the object from a unique direction of view, the parameters comprising parameters defining the positions of a plurality of vertex points in a virtual space and parameters defining relationships between vertex points and hence surface elements of the object, the method comprising, for a surface element:
determining for the said surface element a measure relative to each direction of view, each said measure being representative of the deviation of the surface of the element from the normal to the direction of view;
identifying the direction of view which exhibits the least said deviation;
identifying within that one of the two-dimensional images which corresponds to the identified direction of view a portion thereof corresponding to a projection of the surface element, whereby the image to be generated may be formed having a shape determined by the stored parameters and having surface detail determined by the two-dimensional images.
Preferably, the allocation of surface elements to directions of view is, for selected ones of the elements, predefined and the steps of determining and identifying are carried out only for the remaining surface elements.
The measure may be determined by:
finding the co-ordinates of the vertices of the surface element with respect to a co-ordinate system corresponding to the directions of view;
determining the maximum difference in the co-ordinates in each of the directions of view;
forming ratios of the maximum differences with respect to each direction of view.
Alternatively, the measure may be determined by:
finding the co-ordinates of the vertices of the surface element with respect to a co-ordinate system corresponding to the directions of view;
ascertaining the dimensions of a polyhedron, the faces of which are normal to the respective directions of view and which encloses the vertices of the surface element;
calculating the length of each side of the polyhedron in each direction of view; and
forming ratios of each length with respect to each direction of view.
In another aspect, the invention provides a method of generating an image representing a three-dimensional object, the three-dimensional object being modelled as a stored set, of parameters representing a model of a three-dimensional object and at least two two-dimensional images of the object, each image representing the object from a unique direction of view, the parameters comprising parameters defining the positions of a plurality of vertex points in a virtual space and parameters defining relationships between vertex points and hence surface elements of the object, the method comprising, for a vertex:
identifying those vertices which, together with the said vertex, contribute to representing surface elements which include the said vertex;
determining for the said surface elements a measure relative to each direction of view, each said measure being representative of the deviation of the surfaces of the elements from the normal to the direction of view;
identifying the direction of view which exhibits the least said deviation;
identifying within that one of the two-dimensional images which corresponds to the identified direction of view a portion thereof corresponding to a projection of the surface element, whereby the image to be generated may be formed having a shape determined by the stored parameters and having surface detail determined by the two-dimensional images.
These methods may include the step of adjusting brightness and/or colour values of the surface detail to be applied, such as to reduce discontinuities between a surface element in respect of which one direction of view has been identified and a surface element in repect of which another direction of view has been identified. For example, the adjusting step may comprise:
(a) comparing, for surface elements represented in two or more of said two-dimensional images, brightness and/or colour values of of the surface detail of those elements as represented in the two-dimensional images, and adjusting the values from at least one image in a sense such to reduce the extent to which they differ: and
(b) performing further such adjustments such as to reduce discontiuities between a surface element in respect of which one direction of view has been identified and a immediately adjacent surface element in repect of which another direction of view has been identified.
In yet another aspect of the invention, there is provided an apparatus for generating an image representing a three-dimensional object, comprising:
a store (1) for storing a set of parameters representing a model of a three-dimensional object and at least two two-dimensional images of the object, each image representing the object from a unique direction of view, the parameters comprising parameters defining the positions of a plurality of vertex points in a virtual space and parameters defining relationships between vertex points and hence surface elements of the object; and
control means arranged in operation
to determine for a surface element a measure relative to each direction of view, each said measure being representative of the deviation of the surface of the element from the normal to the direction of view;
to identify the direction of view which exhibits the least said deviation; and
to identify, within that one of the two-dimensional images which corresponds to the identified direction of view a portion thereof corresponding to a projection of the surface element, whereby the image to be generated may be formed having a shape determined by the stored parameters and having surface detail determined by the two-dimensional images.
In a further aspect, there is provided an apparatus for generating an image representing a three-dimensional object, comprising:
a store for storing a set of parameters representing a model of a three-dimensional object and at least two two-dimensional images of the object, each image representing the object from a unique direction of view, the parameters comprising parameters defining the positions of a plurality of vertex points in a virtual space and parameters defining relationships between vertex points and hence surface elements of the object; and
control means arranged in operation
to identify those vertices which, together with the said vertex, contribute to representing a surface element which includes the said vertex;
to determine for the said surface elements a measure relative to each direction of view, each said measure being representative of the deviation of the surfaces of the elements from the normal to the direction of view;
to identify the direction of view which exhibits the least said deviation; and
identify within that one of the two-dimensional images which corresponds to the identified direction of view a portion thereof corresponding to a projection of the surface element, whereby the image to be generated may be formed having a shape determined by the stored parameters and having surface detail determined by the two-dimensional images.
An additional aspect of the invention concerns a method of generating an image representing a three-dimensional object, the three-dimensional object being modelled as a stored set of parameters representing a model of a three-dimensional object and at least two two-dimensional images of the object, each image representing the object from a unique direction of view, the parameters comprising parameters defining the positions of a plurality of vertex points in a virtual space and parameters defining relationships between vertex points and hence surface elements of the object, the method including the generation of an image representation in which the surface detail of each surface element is obtained from one of the said two-dimensional images, and including the step of adjusting brightness and/or colour values of the surface detail to be applied, such as to reduce discontinuities between a surface element in respect of which one direction of view has been identified and a surface element in repect of which another direction of view has been identified.